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SUMMARY: 

This  project  was  a  three  year  continuation  of  ONR  Contract  #N00014-87-K-0780.  It  had 
two  major  aims.  One  was  the  design  and  computer  simulation  of  specific  neural  circuits 
modeled  after  the  early  vision  system  of  higher  vertebrates.  The  second  aim  was  the  design 
and  construction  of  electronic  analog  neural  systems  that  could  be  used  for  implementing 
such  a  vision  system  and  possibly  other  neural  systems.  Our  initial  approach  to  the 
hardware  implementation  was  to  build  an  entirely  hardwired  system  using  VLSI  assemblies 
of  electronic  neurons  and  separate  resistive  synapse  connection  matrices  fabricated  by 
CVD  deposition  of  amorphous  silicon.  During  further  simulation  studies  of  the  vision 
system  it  became  clear  that  at  this  early  state  of  development  a  completely  hardwired 
system  would  be  too  restrictive  and  for  this  reason  we  felt  that  we  needed  a  system  that 
would  incorporate  modifiable  synapses  and  a  programmable  connection  architecture. 
Rather  than  restrict  the  design  of  such  a  system  by  tailoring  it  to  the  specific  architectures 
of  the  vision  system,  we  decided  to  design  a  general  purpose  programmable  neural  analog 
computer  that  would  be  useful  not  only  for  vision  but  also  for  other  tasks  of  real  world 
computation  and  other  problems  suitable  for  neural  computation.  A  prototype  of  this 
machine  was  developed  during  the  past  three  years  and  constructed  in  collaboration  with 
Corticon  INC,  a  small  company  founded  by  the  Principle  Investigators.  The  machine 
performed  as  specified.  A  software  opearting  system  was  developed  and  a  variety  of  simple 
application  programs  were  written  and  run  on  the  machine.  They  included  the  real  time 
decomposition  of  acoustical  patterns,  recognition  of  simple  phonemes  and  examples  of 
optical  pattern  recognition  such  as  OCR.  A  much  larger  machine  suitable  for  commercial 
applications  is  currently  beeing  built. 
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In  addition  to  the  harware  project,  theoretical  work  was  carried  out  on 
fundamentals  of  pattern  analysis  and  recognition  in  Neural  Nets,  in  particular  on  methods 
and  strategies  for  pattern  decomposition  and  representation  as  well  as  one  shot  learning 
algorithms  for  pattern  decoding  useful  in  speech  recognition. 


DETAILED  REPORT: 

Simulations  of  a  Neural  Vision  System 

The  vision  system  simulations  implemented  in  a  simplified  manner  the  basic 
functions  of  a  retina  and  the  primary  visual  cortex  as  far  as  they  are  concerned  with  the 
decomposition  of  static  images  into  visual  primitives  such  as  oriented  edges  and  lines  of 
different  spatial  scales  and  contrast  sign  as  well  as  their  end  points  and  curvature.  The 
system  consisted  of  many  layers  of  neurons  each  layer  tuned  to  specific  primitive  Initial 
simulations  were  performed  on  systems  that  had  only  feed-forward  connections  from  a 
receptor  array  to  arrays  of  ON  CENTER  and  OFF  CENTER  units  and  from  there  to 
orientation  arrays.  During  further  work  it  became  clear  that  certain  problems  such  as  the 
disambiguation  of  contrast  direction  for  oriented  edges  and  improved  orientation  tuning 
required  mutually  inhibitory  connections  between  units  in  different  arrays  and  simulation 
of  such  connections  by  digital  methods  turned  out  to  be  extremely  time  consuming.  Our 
inability  to  explore  the  synaptic  gains  and  architectures  for  these  inhibitory  connections  as 
well  as  the  difficulty  of  determining  an  optimal  architecture  and  connectivity  through 
simulation  were  some  of  the  reasons  for  considering  a  programmable  analog  network. 
Another  reason  came  from  difficulties  in  manufacturing  precise  resistive  networks  with 
available  technology.  Simulations  at  the  available  level  of  accuracy  of  the  synaptic  weights 
showed  that  results  were  marginal  and  that  we  would  either  have  to  spend  considerable 
effort  in  trying  to  achieve  better  resistance  tolerances  or  to  develop  programmable 
synapses  that  would  allow  us  to  tune  each  synapse  individually  and  would  provide  us  with 
means  for  rapidly  exploring  different  connectivities.  These  reasons  as  well  as  our  growing 
appreciation  for  the  need  of  a  programmable  machine  led  us  to  direct  our  hardware  efforts 
towards  the  design  and  implementation  of  a  general  purpose  neural  machine. 


General  Purpose  Neural  Analog  Computer 

The  following  is  a  brief  description  of  the  design.  The  architecture  of  the  machine  is 
loosely  based  on  the  cortex  of  higher  vertebrates,  in  the  sense  that  there  are  sets  of 
individual  neurons  each  of  which  receives  only  a  limited  number  of  inputs,  that  is  not  every 
neuron  is  connected  to  every  other  neuron.  However  in  contrast  to  biological  systems,  our 
machine  would  be  able  to  modify  the  connection  architecture  by  external  control  and  thus 
allow  exploration  of  different  architectures  in  addition  to  adjustment  of  synaptic  weights 


and  neuron  parameters.  Although  the  connections  and  synaptic  gains  are  under  digital 
control  the  machine  runs  in  analog  mode.  A  modular  design  allows  expansion  to  any 
degree  and  at  moderate  to  large  size,  i.e.  1(P  to  10^  neurons,  operational  speed  and  power 
would  exceed  any  currently  available  digital  computer. 

The  machine  contains  large  numbers  of  the  following  separate  elements:  neurons, 
synapses,  routing  switches,  and  connection  lines.  Arrays  of  these  elements  were  fabricated 
on  VLSI  chips  which  are  mounted  on  planar  leadless  chip  carriers  each  of  which  forms  a 
separate  module.  These  modules  are  connected  directly  to  neighboring  modules.  By 
increasing  the  number  of  modules  the  machine  can  be  expanded  to  any  degree.  All 
parameters  such  as  the  connections  between  different  neuron  groups,  individual  synaptic 
gains  and  neuron  parameters,  (threshold  and  time  constant)  are  set  digitally  from  separate 
microprocessor.  In  addition  to  the  analog  outputs  from  the  neurons  which  feed  into  the 
other  neurons,  the  neural  outputs  are  also  multiplexed,  A-D  converted  and  fed  into  a 
digital  computer  that  uses  this  information  in  a  learning  mode  to  set  connections  and 
synaptic  weights. 

The  neuron  design  is  based  on  an  earlier  version  which  we  have  used  in  a  neural 
machine  for  acoustical  pattern  recognition. 


The  synaptic  modules  contain  arrays  of  synapses.  The  synaptic  gain  or  weight  of 
each  synapse  is  set  from  the  microprocessors  by  serial  digital  inputs  which  are  stored  at 
each  synapse.  Dynamic  range  of  the  gain  covers  the  range  from  0  to  10  with  a  logarithmic  5 
bit  resolution,  a  6th  bit  determines  the  sign.  The  synapses  for  the  larger  machine  will  have 
an  8bit  resolution.  Implementation  of  the  weights  is  by  current  mirrors. 


The  switch  modules  serve  to  route  the  signals  between  the  neurons  and  thus 
determine  the  connectivity.  Each  module  contains  a  cross-point  array  of  analog  switches 
which  are  set  by  serial  digital  input.  There  is  also  a  set  of  serial  switches  that  can 
disconnect  selected  input  and  output  lines.  The  line  modules  contain  fixed  lines  that  serve 
as  fillers  between  the  switch  modules. 


All  chips  are  mounted  on  identical  leadless  planar  chip  carriers.  Input  and  output 
lines  of  the  carriers  are  arranged  at  right  angles,  with  identical  leads  on  opposite  sides. 
Groups  of  chip  carriers  are  mounted  on  special  boards  and  connected  directly  to  each 
other. 


Connections,  synaptic  gains  and  time  constants  are  set  by  the  central  computer 
either  from  stored  program  libraries  that  contain  connection  architecture  and  synaptic 
constants  appropriate  for  specific  tasks  or  from  connection  parameters  which  are  computed 
from  neuron  outputs  on  the  basis  of  various  learning  algorithm.  Special  routing  routines 

track  already  occupied  lines,  switches  and  synapses  and  adjust  connection  pathways  _ 

accordingly. 
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Higher  level  control  software,  allowing  the  automatic  transfer  of  conceptual  networks 
into  the  machine  has  also  been  developed.  In  this  case  the  neurons  in  the  conceptual 
network  are  first  partitioned  and  grouped  according  to  the  degree  to  which  their  inputs 
originate  from  common  sources.  The  groups  are  then  assigned  to  specific  modules  in  the 
physical  network  by  a  placement  routine  and  subsequently  interconnected  by  an 
autorouting  program  based  on  min-cut  procedures.  This  program  generates  settings  of 
switches  and  component  parameters  that  are  loaded  into  the  neural  computer. 

The  network  is  connected  directly  to  the  outside  world  through  parallel  analog  I/O 
buffers.  There  are  352  analog  I/O  lines  available.  In  addition  to  the  analog  I/O,  each 
neuron  chip  contains  an  analog  multiplexer  that  enables  the  digital  host  to  monitor  and 
store  the  neuron  activity  for  graphic  display  and  the  implementation  of  programmed 
learning  algorithms.  The  multiplexer  feeds  over  a  common  line  into  a  250  KHz  A/D  board 
located  in  the  host.  The  monitor  software  generates  either  arrays  of  selected  time  segments 
of  each  neuron  output  that  are  displayed  as  separate  graphs,  or  it  generates  a  continuous 
gray  scale  display  of  activity  for  all  neurons.  Neuron  activity  is  also  displayed  directly  by  an 
LED  Panel.  The  neuron  activity  patterns  read  by  the  host  also  form  the  basis  for 
implementation  of  different  learning  algorithms  in  which  the  network  and  the  host  form  a 
closed  loop.  The  learning  of  a  simple  OCR  task  by  backpropagation  has  been 
demonstrated. 

Performance  of  simple  comnutational  tasks. 

The  machine  has  been  programmed  for  several  task.;  in  order  to  evaluate  its 
performance.  A  few  examples  are  discussed.  In  all  cases  the  actual  neural  computations 
(not  the  learning)  are  performed  in  real  time,  with  response  time  limited  only  by  the 
bandwidth  of  the  neurons  which  was  maximally  300  KHz. 

The  first  example,  was  a  "Winner  Take  All"  net  which,  depending  on  the  inhibitory 
feedback  gain,  either  extracts  the  largest  output  among  32  neurons  or  enhances  the 
contrast  between  the  different  outputs.  The  network  settles  within  the  time  constant  of  the 
neuron  outputs  and  shows  no  oscillations.  Simulations  of  this  small  network  on  a  SUN  4 
are  slower  by  a  factor  of  100.  Since  this  is  a  fully  connected  net  the  speed  ratio  would  scale 
with  r?  of  the  number  of  neurons. 

Other  small  circuits,  have  been  programmed,  including  an  associative  net  of  24  neurons, 
a  motor  control  circuit  that  models  the  last  stage  of  the  control  of  saccadic  eye  movements, 
a  neural  integrator  and  several  circuits  for  the  computation  of  time  domain  pattern 
primitives. 

An  example  of  a  time  domain  operation  which  shows  small  circuits  that  are  tuned  to  the 
frequency  of  amplitude  modulation  of  a  carrier  frequency.  Such  circuits  are  useful  in  the 
recognition  of  sonar  patterns.  Several  coupled  oscillators  and  central  pattern  generators 
have  also  been  programmed.  By  making  use  of  threshold  bias,  feedback  and  synaptic  time 
constants,  the  networks  can  generate  arbitrary  dynamic  activity  patterns  in  the  absence  of 
external  inputs. 


The  example  of  character  recognition  demonstrated  that  the  system  can  learn  by 
backpropagation  in  a  simple  4  layer  network.  Development  of  alternative  learning 
algorithms  for  the  machine  is  still  at  an  early  stage.  Various  algorithms  can  be  run  either  by 
the  host  alone  or  with  the  neural  computer  in  loop.  Since  the  weights  are  computed  and 
updated  by  the  host,  the  learning  speed  for  error  correcting  algorithms  is  improved  only  in 
those  cases  where  the  actual  performance  computation  is  the  rate  limiting  factor  such  as  in 
networks  with  extensive  front  ends  or  networks  having  lateral  or  feedback  connections  or 
different  synaptic  time  constants.  However,  we  are  developing  single  shot  learning  schemes 
for  pattern  recognition  where  the  synapse  parameters  are  determined  from  the  input 
patterns  without  iterative  adjustments. 

The  machine  is  especially  suited  for  the  real-time  analysis  of  dynamic  patterns  such  as 
speech.  As  an  example  a  more  elaborate  network  was  programmed  for  the  primary 
decomposition  of  acoustical  patterns  in  real  time  and  the  recognition  of  simple  phonemes. 
This  network,  involving  most  of  the  available  neurons,  performs  an  initial  decomposition  of 
acoustical  patterns.  It  receives  input  from  a  set  of  8  bandpass  filters  (200  -  3000  Hz)  and 
extracts  local  maxima  of  amplitude  vs  frequency,  (d2E/dS2),  local  rates  of  rise  and  fall  of 
amplitude  ( +  dE/Dt),  and  local  rise  and  fall  of  frequency  i.e.  motion  ( +  dS/dT)  of  activity 
along  the  frequency  axis.  The  activity  of  the  decomposition  neurons  were  used  as  inputs  to 
phoneme  specific  neurons  which  became  active  when  a  particular  phonem  was  spoken  to 
the  machine. 

Several  conclusions  can  be  drawn  from  these  tests.  First  of  all  the  machine  performance 
has  met  all  expectations  regarding  noise  levels,  accuracy,  stability,  programming  flexibility 
and  processing  speed.  Particularly  the  routing  space  proved  entirely  adequate.  There  is 
therefore  no  reason  to  project  an  increase  in  the  ratio  of  switch  modules  to  neuron  and 
synapse  modules  for  larger  machines.  No  major  design  change  for  the  neurons  is  needed. 
The  synapse  gain  (weighting)  scheme  could  be  improved  by  increasing  the  resolution  for 
middle  range  gains  (between  0.1  and  1).  The  range  and  resolution  of  synaptic  time 
constants  seems  adequate. 

The  larger  machine  currently  under  development  will  be  capable  of  maximal  speeds  of 
beteen  10“  and  1012  connections/s. 
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